feat: demo mode for showing a trained policy off live#75
Open
ZouzouWP wants to merge 1 commit into
Open
Conversation
Showing a trained policy to someone (recruiters, teammates, a booth) meant re-entering the full inference config (checkpoint, robot, cameras) from scratch every time, with no simple live camera preview separate from the full recording UI.
## What changed
- New `/demo` page: one click loads the **last successfully started inference config** (policy, robot, cameras) as a preset from `localStorage` — no re-configuration needed right before a demo.
- New `lelab/rollout_frames.py`: a thin wrapper around `lerobot.scripts.lerobot_rollout` that runs inference exactly as before, but also tees each camera frame to a JPEG on disk (~10fps). Served through a new `/demo-camera/{cam_id}` endpoint for a live, low-latency preview on the demo page.
```mermaid
flowchart LR
A["InferenceModal:\nstart inference"] -->|saves preset| B["localStorage\npolicy + robot + cameras"]
B --> C["/demo page\none-click preset"]
C --> D["lerobot_rollout subprocess\nvia rollout_frames.py wrapper"]
D -->|tees JPEG ~10fps| E["%TEMP%/lelab_demo_frames"]
E --> F["GET /demo-camera/cam_id"]
F --> G["Live preview\non /demo page"]
```
## Dead end tried and reverted
Real-Time Chunking (`--inference.type=rtc`) was tried to remove a periodic ~200ms stall from ACT's chunk recompute. This lerobot version's RTC engine calls `predict_action_chunk(inference_delay=...)`, which `ACTPolicy` doesn't accept — RTC targets flow-matching policies (pi0, smolvla), not ACT. Reverted cleanly to the standard sync engine; left a comment in the code for whoever revisits ACT-specific prefetching.
## Testing
Run against a checkpoint trained on ~250 episodes: verified the live preview updates (279 frame updates observed in one run) and that the preset correctly restores policy/robot/cameras without manual re-entry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Showing a trained policy to someone (recruiters, teammates, a booth) meant re-entering the full inference config (checkpoint, robot, cameras) from scratch every time, with no simple live camera preview separate from the full recording UI.
What changed
/demopage: one click loads the last successfully started inference config (policy, robot, cameras) as a preset fromlocalStorage— no re-configuration needed right before a demo.lelab/rollout_frames.py: a thin wrapper aroundlerobot.scripts.lerobot_rolloutthat runs inference exactly as before, but also tees each camera frame to a JPEG on disk (~10fps). Served through a new/demo-camera/{cam_id}endpoint for a live, low-latency preview on the demo page.flowchart LR A["InferenceModal:\nstart inference"] -->|saves preset| B["localStorage\npolicy + robot + cameras"] B --> C["/demo page\none-click preset"] C --> D["lerobot_rollout subprocess\nvia rollout_frames.py wrapper"] D -->|tees JPEG ~10fps| E["%TEMP%/lelab_demo_frames"] E --> F["GET /demo-camera/cam_id"] F --> G["Live preview\non /demo page"]Dead end tried and reverted
Real-Time Chunking (
--inference.type=rtc) was tried to remove a periodic ~200ms stall from ACT's chunk recompute. This lerobot version's RTC engine callspredict_action_chunk(inference_delay=...), whichACTPolicydoesn't accept — RTC targets flow-matching policies (pi0, smolvla), not ACT. Reverted cleanly to the standard sync engine; left a comment in the code for whoever revisits ACT-specific prefetching.Testing
Run against a checkpoint trained on ~250 episodes: verified the live preview updates (279 frame updates observed in one run) and that the preset correctly restores policy/robot/cameras without manual re-entry.